Papers with sparse models
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing tasks. |
| Approach: | They propose a parameter-efficient sparsity crafting method which crafts dense models into sparse models using the mixture-of-experts architecture. |
| Outcome: | The proposed method significantly reduces computational costs and GPU memory requirements, while maintaining the quality of approximation in function space. |
From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction (2026.acl-industry)
Copied to clipboard
Egor Maximov, Yulia Kuzkina, Egor Shvetsov, Azamat Kanametov, Aleksandr Prutko, Maxim Zhelnin, Aleksei Goncharov
| Challenge: | Quantization and sparsification are important for large language models, but they struggle to meet performance thresholds due to limited flexibility and sensitivity to outlier weights. |
| Approach: | They propose to use 8:16 semi-structured sparsity to surpass performance thresholds . they also show that structured sparsification for outliers is competitive with unstructured approaches . |
| Outcome: | The proposed method surpasses the Performance Threshold, compared to 2:4 sparsity, and offers greater flexibility with minimal storage overhead. |
Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to information retrieval (IR) and natural language processing (NLP) use contextual language models, which can improve both synonymy and polysemy problems associated with words. |
| Approach: | They propose an ultra-high dimensional representation scheme equipped with directly controllable sparsity and a bucketing method where embeddings from multiple layers of BERT are selected/merged to represent diverse linguistic aspects. |
| Outcome: | The proposed representation scheme outperforms sparse models with MS MARCO and TREC CAR, and shows that it is highly efficient for storage and search. |
On Complementarity Objectives for Hybrid Retrieval (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to hybrid retrieval focus on sparse models to capture “residual” features neglected in spars. |
| Approach: | They propose a new objective to capture a fuller notion of complementarity . they propose to improve the model's Ratio of Complementarity to improve RoC . |
| Outcome: | The proposed method outperforms state-of-the-art methods on three representative IR benchmarks with statistical significance. |
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Sparse attention is a promising strategy to extend long-context capabilities in LLMs . but its efficiency–accuracy trade-offs remain unclear due to the lack of comprehensive evaluation . |
| Approach: | They evaluate sparse attention methods across multiple model families and sizes . they find larger sparser models outperform smaller dense ones at equivalent cost . |
| Outcome: | The proposed methods outperform smaller sparse models at equivalent cost and improve the Pareto frontier. |